Processor for image stabilization based on artificial intelligence and device including the same

ABSTRACT

A method for stabilizing an image based on artificial intelligence includes acquiring tremor detection data with respect to the image, the tremor detection data acquired from two or more sensors; outputting stabilization data for compensating for an image shaking, the stabilization data outputted using an artificial neural network (ANN) model trained to output the stabilization data based on the tremor detection data; and compensating for the image shaking using the stabilization data. A camera module includes a lens; an image sensor to output an image captured through the lens; two or more sensors to output tremor detection data with respect to the image; a controller to output stabilization data based on the tremor detection data using an ANN model; and a stabilization unit to compensate for an image shaking using the stabilization data. The ANN model is trained to output the stabilization data based on the tremor detection data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/885,569, filed on Aug. 11, 2022, which claims the priority of KoreanPatent Application No. 10-2021-0106909 filed on Aug. 12, 2021 and KoreanPatent Application No. 10-2022-0064967 filed on May 26, 2022, in theKorean Intellectual Property Office, the disclosures of which isincorporated herein by reference.

BACKGROUND OF THE DISCLOSURE Technical Field

The present invention relates to a method for image stabilization and acamera module therefor, and particularly, to a method for artificialintelligence-based image stabilization and a camera module therefor.

Background Art

An image captured by a camera may become blurred due to, for example, ashaking (tremor) of a user's hand. To solve this, an image stabilizer(IS) is typically applied to the camera of a camera module to compensatefor camera shake.

An image stabilization technique may use an optical image stabilizer(OIS) and an image sensor. The camera module to which OIS technology isapplied detects vibration (image shake) through a navigation sensor suchas a gyro sensor and generates a compensation signal using a Hall sensorthat detects a change in separation between a coil and a magnet. Basedon the compensation signal, the image shake is stabilized.

In particular, the camera module may compensate for vibration by usingone of a lens shift method, in which the image sensor is fixed and thelens is shifted to compensate for vibration, and a camera rotationmethod in which the image sensor and the lens are simultaneously tiltedto compensate for vibration.

SUMMARY OF THE DISCLOSURE

An OIS system may be implemented as an OIS system to which a lens and animage sensor are both fixed, which is a camera-tilt architecture typeOIS system. Alternatively, an OIS system may be implemented as an OISsystem to which only an image sensor is fixed, which is a barrel-shiftarchitecture type OIS system.

The camera-tilt architecture type OIS system, which may include a photosensor and an actuator, stabilizes image shaking by tilting thestructure to which the lens and image sensor are fixed such that thelens and image sensor are tilted together, i.e., at the same time. Onthe other hand, the barrel-shift architecture type OIS system, which mayinclude an actuator, a Hall sensor, and a magnet, stabilizes imageshaking by shifting a lens using a magnet.

The above OIS systems perform stabilization using a simple feedbackalgorithm that generates a stabilization signal based on the detectionsignals of sensors that detect image shaking.

The inventors of the present disclosure have recognized that ifstabilization is performed using such a simple feedback algorithm, theperformance of the algorithm may be limited.

Accordingly, an object of the present disclosure is to provide anartificial intelligence-based image stabilization method capable ofmaximizing the processing speed of an operation for compensating forimage shaking and reducing power consumption, and to provide a cameramodule using the method.

Another object of the present disclosure is to provide an artificialintelligence-based image stabilization method capable of maximizing theprocessing speed of an operation for compensating for image shaking andreducing power consumption, and to provide a processor using the method.

Another object of the present disclosure is to provide an artificialintelligence-based image stabilization method capable of compensatingfor a plurality of tremor patterns in various usage environments throughartificial intelligence reinforcement learning, and to provide aprocessor using the method.

Another object of the present disclosure is to provide an artificialintelligence-based image stabilization method and an apparatus capableof maximizing the processing speed of an operation for compensating forimage shaking and reducing power consumption through artificialintelligence training.

Another object of the present disclosure is to provide an artificialintelligence-based image stabilization method and an apparatus capableof maximizing the processing speed of an operation for compensating forimage shaking and reducing power consumption by training a specificpattern using an artificial intelligence technique.

The present disclosure is not limited to the above objects, and otherproblems will be clearly understood by those skilled in the art from thefollowing description.

According to one aspect of the present disclosure, a method forstabilizing an image based on artificial intelligence (AI) is provided.The method may include acquiring tremor detection data with respect tothe image, the tremor detection data acquired from two or more sensors;outputting stabilization data for compensating for an image shaking, thestabilization data outputted using an artificial neural network (ANN)model trained to output the stabilization data based on the tremordetection data; and compensating for the image shaking using thestabilization data.

The ANN model may be based on reinforcement learning.

The tremor detection data may include a signal to detect position changeof a camera module and a lens.

The two or more sensors may include a gyro sensor and a Hall sensor.

The ANN model is a trained model so that an error value due to the imageshaking approaches a predetermined value based on training data forlearning.

The ANN model may be configured to output a control signal forcontrolling a movement of a lens included in a camera module tocompensate for the image shaking by receiving the tremor detection dataas an input.

The ANN model may be configured to output a control signal forcontrolling a movement of an image sensor included in a camera module tocompensate for the image shaking by receiving the tremor detection dataas an input.

The ANN model may be configured to receive the tremor detection data asan input to simultaneously perform a training operation for inferencingthe stabilization data and an inference operation regarding to thestabilization data.

The ANN model may include an input node to which the tremor detectiondata is input; a hidden layer for performing an AI operation of theinput node; and an output node to outputting the stabilization data.

According to another aspect of the present disclosure, a camera moduleis provided. The camera module may include a lens; an image sensorconfigured to output an image captured through the lens; two or moresensors configured to output tremor detection data with respect to theimage; a controller configured to output stabilization data based on thetremor detection data using an artificial neural network (ANN) model;and a stabilization unit configured to compensate for an image shakingusing the stabilization data, wherein the ANN model is trained to outputthe stabilization data based on the tremor detection data.

The tremor detection data may include a signal detected by an x-axis andy-axis rotational movement of a gyro sensor and an x-axis and y-axisrotational movement of a Hall sensor.

The two or more sensors may include at least two of a gyro sensor, aHall sensor, and a photo sensor. The ANN model may be trained accordingto the tremor detection data such that an error value due to the imageshaking approaches a predetermined value. The error value may be basedon a difference between the x-axis movement of the gyro sensor and thex-axis movement of the Hall sensor and a difference between the y-axismovement of the gyro sensor and the y-axis movement of the Hall sensor.The trained model may include a first model trained to infer thestabilization data in which the error value approaches the predeterminedvalue, based on the tremor detection data; and a second model trained tocriticize a result of the stabilization data. The controller may befurther configured to collect the error value through the training andto update the ANN model using the collected error value.

The camera module may further include a temperature sensor for sensing atemperature, and the controller may be further configured to output thestabilization data based on the tremor detection data and temperaturedata acquired through the temperature sensor.

The tremor detection data may include a signal detected by an x-axis andy-axis rotational movement of a gyro sensor and by an x-axis, y-axis,and z-axis rotational movement of a Hall sensor, and the controller maybe further configured to obtain a defocus amount data from a frequencycomponent of the image and to output the stabilization data based on thetremor detection data and the defocus amount data.

The ANN model may be configured to use modulation transfer function(MTF) data of the image as training data for training.

According to the present disclosure, by inferring a compensation signalused to compensate for image shaking using an artificial neural network(ANN) model, it is possible to maximize the stabilization speed andreduce power consumption.

In addition, according to the present disclosure, an image stabilizationmethod with improved accuracy can be provided by compensating for imageshaking in consideration of various variables such as temperature,defocus amount, a modulation transfer function (MTF) of the image, andthe like as well as the detection signal of the sensor using anartificial neural network model.

In addition, according to the present disclosure, it is possible toprovide an image stabilization method with improved efficiency byperforming focus adjustment as well as image stabilization.

In addition, according to the present disclosure, image stabilizationcan be quickly provided through a processor that processes a trainedartificial neural network model.

In addition, according to the present disclosure, it is possible toreduce power consumption while rapidly providing image stabilizationthrough an inference dedicated processor that processes a trainedartificial neural network model.

In addition, according to the present disclosure, image stabilizationcan be performed through a plurality of tremor patterns of various usageenvironments trained by artificial intelligence reinforcement learning.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic conceptual diagram illustrating an apparatusincluding a camera module according to an example of the presentdisclosure.

FIG. 2 is a schematic conceptual diagram illustrating a camera moduleaccording to an example of the present disclosure.

FIG. 3 is a schematic conceptual diagram illustrating an artificialneural network model according to an example of the present disclosure.

FIG. 4 is a schematic conceptual diagram illustrating a training methodof an artificial neural network model according to an example of thepresent disclosure.

FIG. 5 is a schematic conceptual diagram illustrating a specifictraining operation of an artificial neural network model according to anexample of the present disclosure.

FIG. 6 is a schematic conceptual diagram illustrating a method ofcompensating for image shaking using a trained artificial neural networkmodel according to an example of the present disclosure.

FIG. 7 is a schematic conceptual diagram illustrating an artificialneural network model according to another example of the presentdisclosure.

FIG. 8 is a schematic conceptual diagram illustrating a training methodof an artificial neural network model according to another example ofthe present disclosure.

FIG. 9 is a schematic conceptual diagram illustrating a method ofcompensating for image shaking using a trained artificial neural networkmodel according to another example of the present disclosure.

FIG. 10 is a schematic conceptual diagram illustrating a camera moduleaccording to another example of the present disclosure.

FIG. 11 is a schematic conceptual diagram illustrating an artificialneural network model according to another example of the presentdisclosure.

FIG. 12 is a schematic conceptual diagram illustrating a training methodof an artificial neural network model according to another example ofthe present disclosure.

FIG. 13 is a schematic conceptual diagram illustrating a specifictraining operation of an artificial neural network model according toanother example of the present disclosure.

FIG. 14 is a schematic conceptual diagram illustrating a method ofcompensating for image shaking using a trained artificial neural networkmodel according to another example of the present disclosure.

FIG. 15 is a flowchart illustrating a method for image stabilization ina camera module in an example of the present disclosure.

FIG. 16 is a schematic conceptual diagram illustrating a neuralprocessing unit according to the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENT

Particular structural or step-by-step descriptions for examplesaccording to the concept of the present disclosure disclosed in thepresent disclosure or application are merely exemplified for the purposeof explaining the examples according to the concept of the presentdisclosure.

Examples according to the concept of the present disclosure may beembodied in various forms, and should not be construed as being limitedto the examples described in the present disclosure or application.

Since the examples according to the concept of the present disclosuremay have various modifications and may have various forms, specificexamples will be illustrated in the drawings and described in detail inthe present disclosure or application. However, this is not intended tolimit the examples according to the concept of the present disclosurewith respect to the specific disclosure form, and should be understoodto include all modifications, equivalents, and substitutes included inthe spirit and scope of the present disclosure.

Terms such as first and/or second may be used to describe variouselements, but the elements should not be limited by the terms.

The above terms are only for the purpose of distinguishing one elementfrom another element, for example, without departing from the scopeaccording to the concept of the present disclosure, and a first elementmay be termed a second element, and similarly, a second element may alsobe termed a first element.

When an element is referred to as being “connected to” or “in contactwith” another element, it is understood that the element may be directlyconnected to or in contact with another element, but other elements maybe disposed therebetween. On the other hand, when it is mentioned that acertain element is “directly connected to” or “directly in contact with”another element, it should be understood that no other element ispresent therebetween.

Other expressions describing the relationship between elements, such as“between” and “immediately between” or “adjacent to” and “directlyadjacent to”, etc., should be interpreted similarly.

In this document, expressions such as “A or B”, “at least one of Aor/and B”, or “one or more of A or/and B” may include all possiblecombinations of the items listed together. For example, “A or B”, “atleast one of A and B”, or “at least one of A or B” can refer to allinstances of (1) including at least one A, (2) including at least one B,or (3) including both at least one A and at least one B.

As used herein, expressions such as “first,” “second,” “first,” or“second,” may modify various elements, regardless of order and/orimportance. In addition, it is used only to distinguish one componentfrom other components, and does not limit the components.

For example, the first user equipment and the second user equipment mayrepresent different user equipment regardless of order or importance.For example, without departing from the scope of rights described inthis document, the first component may be named as the second component,and similarly, the second component may also be renamed as the firstcomponent.

Terms used in present disclosure are only used to describe specificexamples, and may not be intended to limit the scope of other examples.

The singular expression may include the plural expression unless thecontext clearly dictates otherwise. Terms used herein, includingtechnical or scientific terms, may have the same meanings as commonlyunderstood by one of ordinary skill in the art described in thisdocument.

Among terms used in this document, terms defined in a general dictionarymay be interpreted to have the same or similar meaning to the meaning inthe context of the related art. Also, unless explicitly defined in thisdocument, it should not be construed in an ideal or overly formal sense.In some cases, even terms defined in this document cannot be construedto exclude examples of this document.

The terms used herein are used only to describe specific examples, andare not intended to limit the present disclosure.

The singular expression includes the plural expression unless thecontext clearly dictates otherwise. In this specification, terms such as“comprise” or “have” are intended to designate the presence of a statedfeature, number, step, action, component, part, or combination thereof.Accordingly, it should be understood that the existence or addition ofone or more other features or numbers, steps, operations, components,parts, or combinations thereof, is not precluded.

Unless defined otherwise, all terms used herein, including technical orscientific terms, have the same meaning as commonly understood by one ofordinary skill in the art to which this disclosure pertains.

Terms such as those defined in a commonly used dictionary should beinterpreted as having a meaning consistent with the meaning in thecontext of the related art, and should not be interpreted in an ideal orexcessively formal meaning unless explicitly defined in the presentdisclosure.

Each feature of various examples of the present disclosure can bepartially or wholly combined or combined with each other, andtechnically various interlocking and driving are possible as thoseskilled in the art will fully understand. In addition, each example maybe implemented independently of each other or may be implementedtogether in a related relationship.

In describing the examples, descriptions of technical contents that arewell known in the technical field to which the present disclosurepertains and are not directly related to the present disclosure may beomitted. This is to more clearly convey the gist of the presentdisclosure without obscuring the gist of the present disclosure byomitting unnecessary description.

Hereinafter, the present disclosure will be described in detail bydescribing preferred examples of the present disclosure with referenceto the accompanying drawings. Hereinafter, examples of the presentdisclosure will be described in detail with reference to theaccompanying drawings.

FIG. 1 illustrates an apparatus including a camera module according toan example of the present disclosure.

Referring to FIG. 1 , an apparatus A may include a camera module 1000, aprocessor 140, and a first memory 3000. According to various examples,the apparatus A may be configured to further include a second memory4000 separate from the first memory 3000. That is, the apparatus A maybe configured to selectively include or exclude the second memory 4000.

The camera module 1000 is a unit for capturing an image, and may includean OIS system for detecting and compensating for image shaking. Forexample, the OIS system may include two or more sensors for detectingthe image shaking and compensation units for compensating for the imageshaking.

In particular, the OIS system may include a processor 140 that is anarithmetic unit. The processor 140 may be implemented with variousmodifications.

For example, the processor 140 may be a computing device such as acentral processing unit (CPU) or an application processor (AP).

For example, the processor 140 may be a computing device such as a microprocessing unit (MPU).

For example, the processor 140 may be a computing device such as a microcontroller unit (MCU).

For example, the processor 140 may be a computing device such as aneural processing unit (NPU).

For example, the processor 140 may be a computing device such as animage signal processor (ISP).

For example, the processor 140 may be a system on chip (SoC) in whichvarious computing devices such as a CPU and an NPU are integrated.

For example, the processor 140 may be implemented in the form of anintegrated chip (IC) in which the above-described computing devices areintegrated. However, examples of the present disclosure are not limitedto the processor 140.

The processor 140 may operatively communicate with the camera module1000 and the first memory 3000. Additionally, the processor 140 may bein operative communication with the second memory 4000.

The processor 140 may correspond to a computing device such as a CPU oran AP. In addition, the processor 140 may be implemented in the form ofan integrated chip such as an SoC in which various computing devicessuch as CPU, GPU, and NPU are integrated. In various examples, theprocessor 140 may operate as a computing device of the OIS system.

The processor 140 may be implemented in the form of an integrated chipin which various arithmetic devices such as a CPU and an ISP areintegrated. Such an ISP may receive a Bayer patten, which is an outputimage of the camera module, and may output data converted into an RGB orYUV image.

That is, the processor 140 according to examples of the presentdisclosure may refer to a dedicated processor and/or integratedheterogeneous processors.

The camera module 1000 may be implemented in the form of an integratedchip integrated with the processor 140.

The first memory 3000 is a memory mounted on a semiconductor die and maybe a memory for caching or storing data processed in the on-chip region.The first memory 3000 may include a memory such as ROM, SRAM, DRAM,resistive RAM, magneto-resistive RAM, phase-change RAM, ferroelectricRAM, flash memory, or high bandwidth memory (HBM). The first memory 3000may include at least one memory unit. The first memory 3000 may beconfigured as a homogeneous memory unit or a heterogeneous memory unit.

The first memory 3000 may be configured as an internal memory or anon-chip memory.

The second memory 4000 may include a memory such as ROM, SRAM, DRAM,resistive RAM, magneto-resistive RAM, phase-change RAM, ferroelectricRAM, flash memory, or HBM. The second memory 4000 may include at leastone memory unit. The second memory 4000 may be configured as ahomogeneous memory unit or a heterogeneous memory unit.

The second memory 4000 may be configured as a main memory or an off-chipmemory.

Hereinafter, the camera module 1000 will be described in detail withreference to FIG. 2 , which illustrates a camera module according to anexample of the present disclosure.

Referring to FIG. 2 , the camera module 1000 may include a lens 100, animage sensor 110, a first sensor 120, a voice coil motor (VCM) driver130, a processor 140, and a VCM actuator 200.

However, the processor 140 may be included in the camera module 1000, asshown in FIG. 1 , or configured to be disposed outside the camera module1000. That is, the processor 140 of FIG. 2 may be excluded from thecamera module 1000, and the processor 140 may be configured tocommunicate with the camera module 1000 from the outside.

The lens 100 is configured to form an optical image while collecting ordistributing light coming from an object. The image sensor 110 is acomponent that converts light received through a lens into a digitalsignal, and may include a charge-coupled device (CCD) sensor, acomplementary metal oxide semiconductor (CMOS) sensor, and the like.

The first sensor 120 is a sensor for detecting the shaking of the cameramodule 1000 or the apparatus A. The first sensor 120 may be an angularvelocity sensor that measures the rotational speed of an object, such asa gyro sensor. In other words, the movement signal obtained through thefirst sensor 120 may include information on the x-axis movement and they-axis movement of the first sensor 120.

The VCM driver 130 may be configured to transmit a control signal to theVCM actuator 200 to correctly adjust the position of the lens 100.

The VCM actuator 200 may include a lens holder 210, a magnet 220, asecond sensor 230, a coil holder 240, and a coil 250.

Specifically, the lens holder 210 may be configured to mount the lens100, and the magnet 220 may be configured to adjust the position of thelens 100 using a magnetic force.

The second sensor 230 is a sensor for detecting shaking of the VCMactuator 200, and may be a Hall sensor for detecting a change inseparation between the OIS coil 250 and the magnet 220.

The coil holder 240 may be configured such that the OIS coil 250 ismounted on the coil holder 240 and such that the OIS coil 250 isdisposed opposite the magnet 220 to control the magnetic force of themagnet 220.

The processor 140 may be configured to process the overall operation ofthe camera module 1000 in consideration of the lens 100, the imagesensor 110, the first sensor 120, the voice coil motor (VCM) driver 130,and the VCM actuator 200.

In particular, the processor 140 may be configured to perform anoperation for compensating for image shaking using an artificial neuralnetwork (ANN).

Specifically, the processor 140 may obtain a movement signal (i.e., atremor detection signal) from two or more sensors (i.e., at least thefirst sensor 120 and the second sensor 230) included in the cameramodule 1000, respectively. The processor 140 may generate an inferredcompensation signal using an ANN model trained to infer a compensationsignal for compensating for image shaking based on each obtainedmovement signal. The image shaking may be compensated based on thecompensation signal inferred by the processor 140.

Here, the movement signal may include the x-axis value and the y-axisvalue of the Hall sensor together with the x-axis value and the y-axisvalue of the gyro sensor. Furthermore, the processor 140 may furtherinclude an internal memory (not shown) to store various data used tocompensate for image shaking.

According to various examples of the present disclosure, the cameramodule 1000 may be implemented as an edge device including an NPU and anNPU memory.

Hereinafter, a method for compensating for image shaking using an ANNmodel by the processor 140 will be described in detail with reference toFIGS. 3 to 6 .

FIG. 3 illustrates an ANN model according to an example of the presentdisclosure.

Referring to FIG. 3 , the ANN model 300 may receive an input data whichis the tremor detection data 310 including a movement signal of thecamera module 1000 obtained through the first sensor 120 and a movementsignal of the lens 100 obtained through the second sensor 230.

To elaborate, since the image sensor 110 is fixed to the camera module1000, when tremor of the camera module 1000 is sensed through the firstsensor 120, it is possible to substantially detect the tremor of theimage sensor 110. Accordingly, by detecting the tremor of the cameramodule 1000, the tremor of the image sensor 110 may be substantiallysensed.

In detail, since the lens 100 is fixed to the lens holder 210, when thetremor of the lens holder 210 is sensed through the second sensor 230,it is possible to substantially detect the tremor of the lens 100.Accordingly, the shaking of the lens 100 may be substantially sensed bydetecting the shaking of the lens holder 210.

Here, the tremor detection data may include, for example, an x-axisrotation angle and a y-axis rotation angle with respect to the cameramodule 1000, and an x-axis rotation angle and a y-axis rotation anglewith respect to the VCM actuator 200.

In that sense, the tremor detection data of the VCM actuator 200actually means the tremor detection data of the lens 100.

In addition, although it is shown that the lens 100 is included in thecamera module 1000 in FIG. 2 , this is only considering the function ofthe lens 100 in the camera module 1000, and it should be understood thatthe lens 100 is fixed to the lens holder 210.

In the example of the present disclosure, the ANN model 300 may receivethe tremor detection data 310, infer stabilization data for compensatingfor image shaking, and output the inferred stabilization data 320.

The VCM driver 130 may receive the inferred stabilization data 320 andoutput a control signal for stabilizing the shaking of the lens 100. Forexample, the ANN model 300 may further include a conversion unit forconverting the stabilization data 320 into a control signal, and thestabilization data 320 may be converted into a control signal throughthe conversion unit and output.

According to various examples of the present disclosure, thestabilization data 320 may be output as a control signal for controllingthe VCM driver 130 to compensate the position of the image sensor 110based on the inferred stabilization data 320. In an example of thepresent disclosure, the stabilization data 320 may be output as acontrol signal for compensating for positions of the lens 100 and theimage sensor 110.

In the example of the present disclosure, the ANN model 300 may be amodel to be trained by reinforcement learning.

For example, the ANN model 300 may be a model such as RNN, CNN, DNN,MDP, DP, MC, TD (SARSA), QL, DQN, PG, AC, A2C, A3C, and the like.However, the present disclosure is not limited to these and may employvarious ANN models trained to infer position compensation data byinputting tremor detection data as an input.

The ANN model 300 according to various examples of the presentdisclosure may be a model selected from among a plurality of ANN models.The plurality of ANN models may mean an actor model, which will bedescribed later.

Hereinafter, a method of training an ANN model will be described indetail with reference to FIG. 4 , which illustrates a training method ofan ANN model 400 according to an example of the present disclosure. Inthe presented example, it is assumed that the ANN model 400 is a modelbased on reinforcement learning such as Actor-Critic model. Inparticular, the operations to be described below may be performed by theprocessor 140.

Referring to FIG. 4 , the ANN model 400 may be trained to select anoptimal action in a given environment or state.

Here, the ANN model 400 may be based on the Actor-Critic algorithm. TheANN model 400 may include a first model (e.g., actor model) thatdetermines an action when a state S is given in a given environment, anda second model (e.g., critic model) that criticizes the value of thestate S.

A given environment may include the current state St, the nextselectable state St+1, the action a that can be taken under anycondition, the reward r for an action done in a certain state, and thepolicy that determines the probability of taking a particular action ina given state.

Specifically, the first movement signals θYaw and θPitch for the cameramodule 1000 and the second movement signals θHSx and θHSy for the lens100 are obtained through the sensor 410. Here, the first movementsignals θYaw and θPitch may be obtained from the first sensor 120, andthe second movement signals θHSx and θHSy may be obtained from thesecond sensor 230.

For example, the first movement signals θYaw and θPitch may include anx-axis rotation angle θPitch and a y-axis rotation angle θYaw withrespect to the camera module 1000.

For example, the second movement signals θHSx and θHSy may include anx-axis rotation angle θHSx and a y-axis rotation angle θHSy with respectto the lens 100.

Here, the first movement signals θYaw and θPitch are signals forsubstantially detecting the shaking of the image sensor 110. The secondmovement signals θHSx and θHSy are signals for substantially sensing theshaking of the lens 100.

The processor 140 may determine environment data, including currentstate St, next state St+1, action at, reward rt+1, and policy by usingthe obtained tremor detection data (θYaw, θPitch, θHSx, θHSy), and maystore it in the batch memory 420.

Here, the batch memory 420 may be a memory for storing environment datain order to train the ANN model 400 as batch data.

Data for the current state St means the currently acquired tremordetection data (θYaw, θPitch, θHSx, θHSy), and the data for the nextstate St+1 may refer to tremor detection data obtained after taking anaction with respect to the current state. The data for the action atmeans the stabilization data (+Xaxis, −Xaxis, +Yaxis, −Yaxis) of thelens 100 that can be inferred based on the tremor detection data (θYaw,θPitch, θHSx, θHSy), and data for reward rt+1 means an error value. Theerror value may be a value based on a difference value between thex-axis rotation angle θPitch for the camera module 1000 and the x-axisrotation angle θHSx for the lens 100, and a difference value between they-axis rotation angle θYaw for the camera module 1000 and the y-axisrotation angle θHSy for the lens 100.

Subsequently, data regarding the current state St and the next stateSt+1 from the batch memory 420 are input as training data to each of thefirst model (e.g., actor model) and the second model (e.g., criticmodel) of the ANN model 400.

The processor 140 trains the ANN model 400 to determine a policy (i.e.,action) that maximizes the reward rt+1 based on this. Here, the policy(i.e., action) maximizing the reward rt+1 may refer to stabilizationdata in which an error value approaches a predetermined value. Forexample, the predetermined value may be zero. In the present disclosure,when the error value approaches zero, it may be determined that there isno tremor.

Specifically, the first model (e.g., actor model) infers stabilizationdata based on the input tremor detection data (θYaw, θPitch, θHSx,θHSy). The first model (e.g., actor model) determines the probabilitythat the error value approaches a predetermined value when the positionof the lens 100 is compensated based on the inferred stabilization data.

The second model (e.g., critic model) criticizes the value of thestabilization data inferred based on the tremor detection data (θYaw,θPitch, θHSx, θHSy) to which the first model (e.g., actor model) isinput. The criticized result of the value is transmitted to a firstmodel (e.g., actor model) so that the first model (e.g., actor model)can be used to determine a subsequent action.

In the training stage, it is possible to provide the tremor detectiondata in various forms.

For example, the camera module 1000 for reinforcement learning may befixed to a specific jig and programmed to vibrate in a specific pattern.

For example, at least one user may hold the camera module 1000 forreinforcement learning and vibrate it for a specific time.

For example, for reinforcement learning, virtual tremor detection datamay be provided.

Here, the specific pattern may be sitting, walking, running, a motion ofa ship, a motion of a car, a motion of a motorcycle, and the like.

During the reinforcement learning period, at least one specific patternmay be applied.

During reinforcement learning, at least one specific pattern may beapplied sequentially or randomly.

The stabilization data inferred through the first model (e.g., actormodel) is transmitted to the VCM driver 130, and the VCM driver 130transmits a control signal for compensating the vibration of the VCMactuator 200 to the VCM actuator 200.

In the present disclosure, the stabilization data may be converted intovoltages (vx, vy) that can be input to the VCM driver 130 so that theVCM driver 130 controls the VCM actuator 200 and transmitted.

The VCM driver 130 receiving these voltages (vx, vy) transmits a controlsignal for controlling the VCM actuator 200 in the x-axis and the y-axisto the VCM actuator 200. Here, the control signal may include a currentcx for controlling the VCM actuator 200 in the x-axis and a current cyfor controlling the VCM actuator 200 in the y-axis. By compensating theposition of the lens 100 by the VCM actuator 200 receiving the currentscx and cy, image shaking can be compensated.

In the next step, tremor detection data may be acquired through thesensor 410, and the above-described training operations may berepeatedly performed. These training operations may be performed until asuccess criterion or a maximum number of epochs is reached, but is notlimited thereto. For example, the success criterion may include acriterion in which an error value determined based on inferredstabilization data approaches zero.

According to various examples of the present disclosure, the processor140 may update the ANN model 400 by collecting the error values obtainedduring the training operation and further training the artificial neuralnetwork model 400 using the collected error values.

According to various examples of the present disclosure, the processor140 may acquire modulation transfer function (MTF) data of an image andtrain a second model (e.g., critic model) based on the acquired MTFdata.

The first model (e.g., actor model) according to various examples of thepresent disclosure may be customized through reinforcement learning.

Hereinafter, a detailed learning operation of the ANN model 400 will bedescribed in detail with reference to FIG. 5 , which illustrates aspecific training operation of an ANN model according to an example ofthe present disclosure.

Referring to FIG. 5 , each of the first model 500 and the second model510 receives the currently acquired tremor detection data (θYaw, θPitch,θHSx, θHSy) st and the tremor detection data st+1 obtained after takingan action on the current state as training data.

The first model 500 receives the tremor detection data (st, st+1) as aninput and outputs an action and a policy. Here, the action meansinferred stabilization data, and the policy includes a probabilityπθ(at|St) of taking an action at in the current state St, and aprobability πθold(at|St) of taking an action at in the current state Stof the first model 500 before being updated to batch.

The second model 510 receives the tremor detection data (st, st+1) as aninput, and outputs the values Vν(St) and Vν(St+1) for the tremordetection data (st, st+1), and the expected profits (e.g., advantage)Â_(t) with respect to the output values Vν(St) and Vν(St+1). Here, thevalue Vν(St) means the value of the tremor detection data st in thecurrent state, and the value Vν(St+1) means the value of the tremordetection data st+1 in the next state.

The first model 500 may determine the loss L^(CLIP)(0) based onπθ(at|St), πθold(at|St), and the output Â_(t) of the second model 510.Here, θ may mean a parameter vector indicating all weights and biases ofthe first model 500. The loss L^(CLIP)(0) can be expressed by thefollowing Equation 1.

L ^(CLIP)(θ)=−min{f _(t)(θ)Â _(t),clip(f _(t)(θ),1−ε,1+ε)Â_(t)}  Equation 1

Here, f_(c)(θ) may mean the quotient ofπθ(a_(t)|s_(t))/πθold(a_(t)|s_(t)), and A_(t) may meanλ{(r+γV_(u)(S_(t)))−V_(u)(S_(t+1))} where r denotes a reward and γdenotes a discount factor.

The first model 500 may update the parameter vector θ of the first model500 using the aforementioned loss L^(CLIP)(θ).

The second model 510 may determine a loss L^(v)(u) using the valuesVν(S_(t)) and Vν(S_(t)+1). Here, ν may mean a parameter vector of thesecond model 510. The loss L^(v)(u) may be expressed by the followingEquation 2.

L ^(v)(u)=V _(u)(s _(t+1))−(r+γV _(u)(s _(t)))  Equation 2

The second model 510 may update the parameter vector ν of the secondmodel 510 by using the above-described loss L^(v)(u).

The inferred stabilization data, which is the action transferred fromthe first model 500, is changed into voltages vx and vy that can beinput from the VCM driver 130 through the conversion unit 520. Forexample, the stabilization data may be converted into an analog voltageor digital signal for controlling the VCM driver 130. When the voltageoutput from the conversion unit 520 is switched, overshooting or ringingmay occur. In order to solve this problem, the conversion unit 520 mayuse an ANN model trained so that the voltage is sloped (e.g., damped orboosted) so as to compensate for the overshooting or ringing and output.

The VCM driver 130 transmits the control signals cx and cy so that theVCM actuator 200 compensates the position of the lens, and thereafter, atraining operation may be performed as described with respect to FIG. 4.

Hereinafter, a method of compensating for image shaking using an ANNmodel trained as described with reference to FIGS. 4 and 5 will bedescribed in detail with reference to FIG. 6 .

In the trained ANN model, weight data is no longer updated in a state inwhich weight data has been trained.

In more detail, the processor 140 may be a plurality of processors suchthat performing training and performing inference may be respectivelyperformed using different processors.

For example, the processor 140 performing training according to examplesof the present disclosure may be a GPU, and the processor 140 performinginference may be an NPU. That is, the processor used in the machinelearning process and the processor used in the inference process afterthe machine learning is completed may be different from each other.

For example, the processor 140 implemented as an NPU may be a dedicatedprocessor for inference operation with a relatively fast processingspeed and reduced power consumption but not supporting the machinelearning algorithm.

The processor 140 implemented as an NPU is not for training, but it isimplemented as a high-speed, low-power processor 140 and has theadvantage that it can be implemented as a variety of edge devices. Inaddition, by performing additional machine learning with the processor140 capable of machine learning and then providing the updated weightdata to the processor 140 implemented as an NPU, it is possible to solvethe problem of unsupported machine learning algorithm.

In more detail, the first model (e.g., actor model) according toexamples of the present disclosure may be configured to be processed bythe processor 140 implemented as an NPU built in the camera module 1000.

In more detail, the second model (e.g. critic model) according toexamples of the present disclosure may be configured to be processed bythe training-only processor 140. Here, the training-only processor 140may be a separate processor disposed outside the camera module 1000.However, examples of the present disclosure are not limited thereto.

FIG. 6 illustrates a method of compensating for image shaking using atrained ANN model according to an example of the present disclosure.Operations to be described below may be performed through the processor140 implemented as an NPU. However, the processor 140 according to theexamples of the present disclosure is not limited to the NPU.

In the example of FIG. 6 , the second model (e.g., critical model) maybe excluded. Accordingly, inference can be performed using only thefirst model (e.g., actor model). In this case, training of the weight ofthe first model (e.g., actor model) may be completed. In addition, whenonly the first model (e.g., actor model) is used, the training step maybe excluded and the second model (e.g., critic model) may be excluded,so that power consumption, calculation amount, and processing speed maybe improved. In addition, a high-speed and low-power camera module 1000may be implemented by applying the low-power processor 140 implementedas an NPU.

Referring to FIG. 6 , the tremor detection data St corresponding to theenvironment is obtained through the sensor 600, and the obtained tremordetection data St is stored in the batch memory 610.

The tremor detection data St stored in the batch memory 610 is input asinput data of the stabilization signal generator 620. The stabilizationsignal generator 620 may include, for example, a first model (e.g.,actor model) including four input nodes, a plurality of hidden layers,and four output nodes. However, the structure of the first model (e.g.,actor model) is not limited thereto.

When (θYaw, θPitch, θHSx, θHSy) are input from each of the four inputnodes, the first model (e.g., actor model) infers stabilization data(+Xaxis, —Xaxis, +Yaxis, —Yaxis) for convergence of the error value r toa predetermined value. Accordingly, the first model (e.g., actor model)may output the inferred stabilization data (+Xaxis, —Xaxis, +Yaxis,—Yaxis).

Here, the error value r may be a summed value of a difference value(errx) between the x-axis rotation angle θPitch for the camera module1000 and the x-axis rotation angle θHSx for the lens 100, and adifference value (erry) between the y-axis rotation angle θYaw for thecamera module 1000 and the y-axis rotation angle θHSy for the lens 100.For example, the error value may be expressed as—(| errx+erry |).

The conversion unit 520 may change the stabilization data (+Xaxis,—Xaxis, +Yaxis, —Yaxis) inferred from the first model (e.g., actormodel) into a voltage available for input of the VCM driver 130. Theconversion unit 520 converts the stabilization data into voltages vx andvy using a conversion formula or a lookup table, and outputs theconverted voltages vx and vy as control signals. For example, thesevoltages (vx, vy) are in a range between 0 V to 5 V, and a correspondingrange may correspond to the input standard of the VCM driver 130.

For example, when +Xaxis is input, the voltage of the stabilizationvoltage vx may be increased by one step, for example, 0.001 V. Forexample, when −Xaxis is input, the voltage of the stabilization voltagevx may be decreased by one step, for example, 0.001 V.

For example, when +Yaxis is input, the voltage of the stabilizationvoltage vy may be increased by one step, for example, by 0.001 V. Forexample, when −Yaxis is input, the voltage of the stabilization voltagevy may be decreased by one step, for example, by 0.001 V. That is, thestabilization voltage may be varied in units of preset voltage steps.The stabilization speed may be determined according to the unit of thevoltage step and the interval at which the voltage step is updated.

However, examples of the present disclosure are not limited to thevoltage step and the time interval.

When voltages vx and vy are input, the VCM driver 130 may output acontrol signal for compensating for tremor of the VCM actuator 200according to stabilization data. This control signal may be currents(cx, cy) for controlling the VCM actuator 200. For example, the currents(cx, cy) may be −100 mA to +100 mA, and the corresponding range maycorrespond to the input standard of the VCM actuator 200.

When the currents (cx, cy) are input, the VCM actuator 200 operates tocompensate the position of the lens 100 according to the input current,thereby performing image stabilization.

Hereinafter, a method in which the apparatus A further includes a sensorfor measuring temperature and compensates for image shaking by furtherconsidering temperature data will be described in detail with referenceto FIGS. 7 to 9 .

FIG. 7 illustrates an ANN model according to another example of thepresent disclosure.

Referring to FIG. 7 , an ANN model 700 includes the movement signal ofthe camera module 1000 obtained through the first sensor 120 and themovement signal of the lens 100 obtained through the second sensor 230.The included tremor detection data 710 and the temperature data 720obtained through the temperature sensor are input as input data.

In more detail, since the image sensor 110 is fixed to the camera module1000, when vibration of the camera module 1000 is sensed through thefirst sensor 120, it is possible to substantially detect the vibrationof the image sensor 110. Accordingly, the tremor of the image sensor 110may be substantially sensed by detecting the tremor of the camera module1000.

In more detail, since the lens 100 is fixed to the lens holder 210, whenthe vibration of the lens holder 210 is sensed through the second sensor230, the vibration of the lens 100 can be substantially sensed.Accordingly, the tremor of the lens 100 may be substantially sensed bydetecting the tremor of the lens holder 210.

Here, the tremor detection data may refer to the tremor detection datadescribed above with reference to FIG. 3 . The temperature sensor formeasuring the temperature data may be a sensor included in the cameramodule 1000, a sensor included in the processor 140, or a dedicatedsensor, but is not limited thereto.

In the example of the present disclosure, the ANN model 700 may inferstabilization data for compensating for image shaking by inputting thetremor detection data 710 and the temperature data 720 as inputs, andthe ANN model may output the inferred stabilization data 730.

Hereinafter, the training method of the ANN model 700 of FIG. 7 will bedescribed in detail with reference to FIG. 8 .

FIG. 8 illustrates a training method of an ANN model according toanother example of the present disclosure. In the presented example, itis assumed that the ANN model 700 is a model based on reinforcementlearning such as Actor-Critic. In particular, the operations to bedescribed below may be performed by the processor 140.

Referring to FIG. 8 , the first movement signals θYaw and θPitch for thecamera module 1000, the second movement signals θHSx and θHSy for thelens 100, and the temperature data Temp are obtained through the sensor710.

Here, the first movement signals θYaw and θPitch are obtained from thefirst sensor 120, the second movement signals θHSx and θHSy are obtainedfrom the second sensor 230, and a temperature signal Temp is obtainedfrom the temperature sensor.

Here, the first movement signals θYaw and θPitch are signals forsubstantially detecting the shaking of the image sensor 110. The secondmovement signals θHSx and θHSy are signals for substantially sensing theshaking of the lens 100.

The processor 140 may determine and store environmental data including acurrent state St, a next state St+1, an action at, a reward rt+1, and apolicy in the batch memory 820 by using the obtained tremor detectiondata (θYaw, θPitch, θHSx, θHSy) and temperature data Temp.

Here, the data on the current state St may mean the currently acquiredtremor detection data (θYaw, θPitch, θHSx, θHSy) and the temperaturedata Temp. The data for the next state St+1 may refer to vibrationdetection data and temperature data obtained after taking an action withrespect to the current state.

The data with respect to action at means stabilization data (+Xaxis,—Xaxis, +Yaxis, —Yaxis) of the lens 100 that can be inferred based onthe tremor detection data (θYaw, θPitch, θHSx, θHSy) and the temperaturedata Temp.

Data for reward rt+1 means an error value based on the temperature dataTemp and a difference value. The difference value is a differencebetween two values. Here, a first value is a difference between thex-axis rotation angle θPitch with respect to the camera module 1000 andthe x-axis rotation angle θHSx with respect to the lens 100, and asecond value is a difference between the y-axis rotation angle θYaw withrespect to the camera module 1000 and the y-axis rotation angle θHSywith respect to the lens 100.

Subsequently, data regarding the current state St and the next stateSt+1 from the batch memory 820 are input to each of the first model(e.g., actor model) and the second model (e.g., critic model) of the ANNmodel 400 as training data.

The processor 140 trains the artificial neural network model todetermine a policy (i.e., action) that maximizes the reward rt+1 basedon this. Specifically, the first model (e.g., actor model) infersstabilization data based on the input tremor detection data (θYaw,θPitch, θHSx, θHSy) and the temperature data Temp. The first model(e.g., actor model) determines the probability that the error valueapproaches a predetermined value when the position of the lens 100 iscompensated based on the inferred stabilization data.

The second model (e.g., critic model) criticizes the value of thestabilization data inferred based on the vibration detection data (θYaw,θPitch, θHSx, θHSy) and the temperature data Temp to which the firstmodel (e.g., actor model) is input.

The criticized result of the value from the second model is transmittedto a first model (e.g., actor model) so that the first model (e.g.,actor model) can be used to determine a subsequent action.

In the training stage, it is possible to provide the tremor detectiondata in various forms.

For example, the camera module 1000 for reinforcement learning may befixed to a specific jig and programmed to move in a specific pattern.

For example, at least one user may hold the camera module 1000 forreinforcement learning and move it for a specific time.

For example, for reinforcement learning, virtual tremor detection datamay be provided.

Here, the specific pattern may be sitting, walking, running, a motion ofa ship, a motion of a car, a motion of a motorcycle, and the like.

During the reinforcement learning period, at least one specific patternmay be applied.

During reinforcement learning, at least one specific pattern may beapplied sequentially or randomly.

The stabilization data inferred through the first model (e.g., actormodel) is transmitted to the VCM driver 130, and the VCM driver 130transmits a control signal for compensating the vibration of the VCMactuator 200 to the VCM actuator 200. The VCM driver 130 receiving thevoltages vx and vy transmits a control signal for controlling the VCMactuator 200 in the x-axis and the y-axis to the VCM actuator 200. Bycompensating the position of the lens 100 by the VCM actuator 200receiving the control signal, image shaking can be compensated.

In the next step, tremor detection data and temperature data may beacquired through the sensor 810, and the above-described trainingoperations may be repeatedly performed. These training operations may beperformed until a success criterion or a maximum number of epochs isreached, but is not limited thereto. For example, the success criterionmay include a criterion in which an error value determined based oninferred stabilization data approaches zero.

Hereinafter, a detailed training operation of the ANN model 800 will bedescribed in detail with reference to FIG. 5 . The ANN model 800 mayperform training in the same way as described with reference to FIG. 5 .

Referring to FIG. 5 , each of the first model 500 and the second model510 receives a current state St including the currently acquired tremordetection data (θYaw, θPitch, θHSx, θHSy) and temperature data Temp, anda next state st+1 including the tremor detection data and temperaturedata obtained after taking an action on the current state as trainingdata.

The first model 500 outputs stabilization data inferred by inputtingtremor detection data and temperature data (st, st+1) along with theprobability of taking action πθ(at|St) in the current state St and theprobability of taking action πθold(at|St) in the current state St of thefirst model 500 before being updated to batch.

The second model 510 receives the tremor detection data and temperaturedata (st, st+1) as an input, and outputs the values Vν(St) and Vν(St+1)for the tremor detection data (st, st+1), and the expected profits(e.g., advantage) A.

The first model 500 may determine the loss L^(CLIP)(0) based onπθ(at|St), πθold(at|St) and the output Â_(t) of the second model 510,and by using the determined loss L^(CLIP)(θ), the parameter vector θ ofthe first model 500 can be updated. The loss L^(CLIP)(θ) can be updatedby using above Equation 1.

The second model 510 may determine a loss L^(v)(u) using the valuesVν(St) and Vν(St+1) and by using the determined loss L^(v)(u), theparameter vector ν of the second model 510 may be updated. The loss canbe calculated using the above Equation 2.

The stabilization data transmitted from the first model 500 is convertedinto voltages (vx, vy) that can be input by the VCM driver 130 throughthe conversion unit 520 and output, the VCM driver 130 transmits thecontrol signals (cx, cy) so that the VCM actuator 200 compensates forthe position of the lens, and thereafter, a training operation may beperformed as described in FIG. 8 .

Hereinafter, a method of compensating for image shaking using the ANNmodel trained as described with reference to FIGS. 5 and 8 will bedescribed in detail with reference to FIG. 9 .

FIG. 9 illustrates a method of compensating for image shaking using atrained ANN model according to another example of the presentdisclosure. Operations to be described later may be performed throughthe processor 140.

In the example of FIG. 9 , the second model (e.g., critical model) maybe excluded. Accordingly, inference can be performed using only thefirst model (e.g., actor model). In this case, training of the weight ofthe first model (e.g., actor model) may be completed.

In addition, when only the first model (e.g., actor model) is used, thetraining step may be excluded and the second model (e.g., critic model)may be excluded, so that power consumption, calculation amount, andprocessing speed may be improved. In addition, a high-speed, low-powercamera module 1000 may be implemented by applying the low-powerprocessor 140 implemented as an NPU.

Referring to FIG. 9 , tremor detection data and temperature data Stcorresponding to the environment are obtained through the sensor 900,and the obtained tremor detection data and temperature data St arestored in the batch memory 910.

The tremor detection data and temperature data St stored in the batchmemory 910 are input as input data of the stabilization signal generator920. The stabilization signal generator 920 may include a first model(e.g., actor model) including five input nodes, a plurality of hiddenlayers, and four output nodes. However, the structure of the first model(e.g., actor model) is not limited thereto.

When θYaw, θPitch, θHSx, θHSy, and Temp is inputted from each of thefive input nodes, the first model (e.g., actor model) infersstabilization data (+Xaxis, —Xaxis, +Yaxis, —Yaxis) for convergence ofthe error value r to a predetermined value, and outputs the inferredstabilization data (+Xaxis, —Xaxis, +Yaxis, —Yaxis). Here, the errorvalue r is a summed value of a difference value errx between the x-axisrotation angle θPitch for the camera module 1000 and the x-axis rotationangle θHSx for the lens 100, and a difference value erry between they-axis rotation angle θYaw for the camera module 1000 and the y-axisrotation angle θHSy for the lens 100. For example, the error value maybe expressed as—(| errx+erry |).

The stabilization signal generator 920 may further include a conversionunit Transpose configured to change the inferred stabilization data(+Xaxis, —Xaxis, +Yaxis, —Yaxis) to a voltage available for input of theVCM driver 130. The conversion unit Transpose converts the stabilizationdata into voltages (vx, vy) using a conversion formula or a lookuptable, and outputs the converted voltages (vx, vy) as control signals.

When voltages (vx, vy) are input, the VCM driver 130 may output acontrol signal for compensating for tremor of the VCM actuator 200according to stabilization data. These control signals may be currents(cx, cy) for controlling the VCM actuator 200.

When the currents (cx, cy) are input, the VCM actuator 200 operates tocompensate the position of the lens 100 according to the input current,thereby performing image stabilization.

Hereinafter, the apparatus A may further include a coil for auto focus(AF), and a method of controlling AF and OIS based on an ANN will bedescribed in detail with reference to FIGS. 10 to 14 .

FIG. 10 illustrates a camera module according to another example of thepresent disclosure. In the presented example, descriptions of redundantelements may be omitted for convenience of description.

Referring to FIG. 10 , the camera module 1000 may include a lens 100, animage sensor 110, a first sensor 120, a VCM driver 130, a processor 140and a VCM actuator 200. The VCM actuator 200 may further include an AFcoil 260.

The AF coil 260 may be disposed to correspond to an AF magnet (notshown), and a voltage for controlling the position of the AF magnet maybe applied.

The processor 140 may obtain a movement signal from two or more sensors(i.e., at least the first sensor 120 and the second sensor 230) includedin the camera module 1000, may determine the amount of defocus from thefrequency component of the image, and may infer the tremor compensationsignal using the ANN model trained to infer the tremor compensationsignal for compensating for image tremor and focus. Subsequently, theprocessor 140 may compensate for image shaking and focus based on theinferred tremor compensation signal. Here, the movement signal mayfurther include a z-axis value of the Hall sensor for focus adjustment.

Hereinafter, a method for compensating for image shaking by using an ANNmodel in the processor 140 and adjusting focus will be described indetail with reference to FIGS. 11 to 14 .

FIG. 11 illustrates an ANN model according to another example of thepresent disclosure.

Referring to FIG. 11 , in an ANN model 1100, a tremor detection data1110 including a movement signal of the camera module 1000 obtainedthrough the first sensor 120 and a movement signal of the lens 100obtained through the second sensor 230 and a defocus amount data 1120determined from the frequency component of the image are input as inputdata. Here, the tremor detection data may further include a z-axisrotation angle obtained through the Hall sensor.

In the example of the present disclosure, the artificial neural networkmodel 1100 may compensate for image shaking by receiving tremordetection data 1110 and defocus amount data 1120 as inputs, inferstabilization data for adjusting a focus, and may output the inferredstabilization data 1130.

Hereinafter, a training method of the ANN model 1100 of FIG. 11 will bedescribed in detail with reference to FIG. 12 .

FIG. 12 illustrates a training method of an ANN model according toanother example of the present disclosure. In the presented example, itis assumed that the ANN model 1200 is a model based on reinforcementlearning such as Actor-Critic. In particular, operations to be describedbelow may be performed through the processor 140.

Referring to FIG. 12 , a first movement signal (θYaw, θPitch) for thecamera module 1000 and a second movement signal (θHSx, θHSy, θHSz) forthe lens 100 are obtained through the sensor 1210, and defocus amountdata determined from the frequency component of the image is obtained.The defocus amount data may be acquired by the processor 140, an imagesensor, or an image signal processor (ISP).

The processor 140 may determine environmental data including a currentstate S_(t), a next state S_(t)+1, an action at, a reward rt+1, and apolicy, by using the obtained tremor detection data (θYaw, θPitch, θHSx,θHSy), the lens focus data θHSz, and the defocus amount data defocusamount, and then the determined environment data may be stored in thebatch memory 1220. Here, the data on the current state S_(t) may meancurrently acquired tremor detection data (θYaw, θPitch, θHSx, θHSy),lens focus data θHSz, and defocus amount data defocus amount. Data forthe next state S_(t)+1 may mean tremor detection data and defocus amountdata obtained after taking an action on the current state. The data forthe action at may mean stabilization data (+Xaxis, −Xaxis, +Yaxis,−Yaxis, +Zaxis, −Zaxis,) of the lens 100 that can be inferred based onthe tremor detection data (θYaw, θPitch, θHSx, θHSy), the lens focusdata θHSz, and the defocus amount data defocus amount. The data for thereward rt+1 may mean an error value based on defocus amount data and adifference value, where the difference value is a difference between twovalues. Here, a first value is a difference between the x-axis rotationangle θPitch for the camera module 1000 and the x-axis rotation angleθHSx for the lens 100, and a second value is a difference between they-axis rotation angle θYaw for the camera module 1000 and the y-axisrotation angle θHSy for the lens 100.

Subsequently, data for the current state St and the next state St+1 fromthe batch memory 1220 are input to each of the first model (e.g., actormodel) and the second model (e.g., critic model) of the ANN model 1200as training data.

The processor 140 trains the ANN model 1200 based on this to infer thestabilization data in which the error value approaches a predeterminedvalue.

Specifically, the first model (e.g., actor model) infers stabilizationdata based on the input shake detection data (θYaw, θPitch, θHSx, θHSy),the lens focus data θHSz, and the defocus amount data defocus amount.The first model (e.g., actor model) determines the probability that theerror value approaches a predetermined value when the position of thelens 100 is compensated based on the inferred stabilization data.

The second model (e.g., critic model) criticizes the value of thestabilization data inferred based on the tremor detection data (θYaw,θPitch, θHSx, θHSy), the lens focus data (θHSz), and the defocus amountdata (defocus amount) inputted by the first model (e.g., actor model).The criticized result of the value is transmitted to a first model(e.g., actor model) so that the first model (e.g., actor model) can beused to determine a subsequent action.

In the training stage, it is possible to provide the tremor detectiondata in various forms.

For example, the camera module 1000 for reinforcement learning may befixed to a specific jig and programmed to vibrate in a specific pattern.

For example, at least one user may hold the camera module 1000 forreinforcement learning and vibrate it for a specific time.

For example, for reinforcement learning, virtual tremor detection datamay be provided.

Here, the specific pattern may be sitting, walking, running, a motion ofa ship, a motion of a car, a motion of a motorcycle, and the like.

During the reinforcement learning period, at least one specific patternmay be applied.

During reinforcement learning, at least one specific pattern may beapplied sequentially or randomly.

The stabilization data inferred through the first model (e.g., actormodel) is transmitted to the VCM driver 130, and the VCM driver 130transmits a control signal for compensating the vibration of the VCMactuator 200 to the VCM actuator 200. In the present disclosure, thestabilization data may be converted into voltages (vx, vy, vz) that canbe input to the VCM driver 130 so that the VCM driver 130 controls theVCM actuator 200 and transmitted. The VCM driver 130 receiving thesevoltages (vx, vy, vz) transmits a control signal for controlling the VCMactuator 200 in the x-axis, y-axis, and z-axis to the VCM actuator 200.Here, the control signal may include a current cx for controlling theVCM actuator 200 in the x-axis, a current cy for controlling the VCMactuator 200 in the y-axis, and a current cz for controlling the VCMactuator 200 in the z-axis.

The VCM actuator 200 receiving these currents cx, cy, and cz operates sothat the position of the lens 100 is compensated, thereby compensatingfor image shaking.

In the next step, after the tremor detection data (θYaw, θPitch, θHSx,θHSy) and lens focus data θHSz are acquired through the sensor 1210, andthe defocus amount data defocus amount is obtained from the frequencycomponent of the image, the above-described training operations may berepeatedly performed. These training operations may be performed until asuccess criterion or a maximum number of epochs is reached, but is notlimited thereto. For example, the success criterion may include acriterion in which an error value determined based on inferredstabilization data approaches zero.

Hereinafter, a specific training operation of the ANN model 1200 will bedescribed in detail with reference to FIG. 13 , which illustrates aspecific training operation of an ANN model according to another exampleof the present disclosure.

Referring to FIG. 13 , each of the first model 1300 and the second model1310 receives the current state st including currently acquired tremordetection data (θYaw, θPitch, θHSx, θHSy), lens focus data θHSz, anddefocus amount data defocus amount, and the next state st+1 includingtremor detection data and defocus amount data obtained after taking anaction on the current state as training data.

The first model 1300 outputs stabilization data inferred by inputtingtremor detection data and defocus amount data (st, st+1) along with theprobability of taking action πθ(at|St) in the current state St and theprobability of taking action πθold(at|St) in the current state St of thefirst model 1300 before being updated to batch.

The second model 1310 receives the tremor detection data and defocusamount data (st, st+1) as an input, and outputs the values Vν(St) andVν(St+1) for the tremor detection data (st, st+1), and the expectedprofits (e.g., advantage) A.

The first model 1300 may determine the loss L^(CLIP)(θ) based onπθ(at|St), πθold(at|St), and the output Â_(t) of the second model 1310,and by using the determined loss L^(CLIP)(θ), the parameter vector θ ofthe first model 500 can be updated. The loss L^(CLIP)(θ) can be updatedby using the above Equation 1.

The second model 1310 may determine a loss L^(v)(u) using the valuesVν(St) and Vν(St+1) and by using the determined loss L^(v)(u), theparameter vector ν of the second model 1310 may be updated. The loss canbe calculated using the above Equation 2.

The stabilization data transmitted from the first model 1300 isconverted into voltages (vx, vy, vz) that can be input by the VCM driver130 through the conversion unit 1320 and output, the VCM driver 130transmits the control signals (cx, cy, cz) so that the VCM actuator 200compensates for the position of the lens, and thereafter, a trainingoperation may be performed as described in FIG. 12 .

Hereinafter, a method of compensating for image shaking using the ANNmodel trained as described with reference to FIGS. 12 and 13 will bedescribed in detail with reference to FIG. 14 .

FIG. 14 illustrates a method of compensating for image shaking using atrained ANN model according to another example of the presentdisclosure. Operations to be described later may be performed throughthe processor 140.

Referring to FIG. 14 , tremor detection data is obtained through thesensor 1400, and after defocus amount data S_(t) is obtained from thefrequency component of the image, the obtained tremor detection data anddefocus amount data S_(t) are stored in the batch memory 1410.

In the example of FIG. 14 , the second model (e.g., critical model) maybe excluded. Accordingly, inference can be performed using only thefirst model (e.g., actor model). In this case, training of the weight ofthe first model (e.g., actor model) may be completed. In addition, whenonly the first model (e.g., actor model) is used, the training step maybe excluded and the second model (e.g., critic model) may be excluded,so that power consumption may be reduced, calculation amount may bereduced, and processing speed may be increased. In addition, ahigh-speed, low-power camera module 1000 may be implemented by applyingthe low-power processor 140 implemented as an NPU.

The tremor detection data and defocus amount data St stored in the batchmemory 1410 are input as input data of the stabilization signalgenerator 1420. The stabilization signal generator 1420 may include afirst model (e.g., actor model) including six input nodes, a pluralityof hidden layers, and six output nodes. However, the structure of thefirst model (e.g., actor model) is not limited thereto.

When θYaw, θPitch, θHSx, θHSy, θHSz, and defocus amount is inputted fromeach of the six input nodes, the first model (e.g., actor model) infersstabilization data (+Xaxis, —Xaxis, +Yaxis, —Yaxis, +Zaxis, —Zaxis) forconvergence of the error value r to a predetermined value, and outputsthe inferred stabilization data (+Xaxis, —Xaxis, +Yaxis, —Yaxis, +Zaxis,—Zaxis). Here, the error value r may be a value obtained by adding adefocus amount (defocus amount) to a summed value of a difference valueerrx between the x-axis rotation angle θPitch for the camera module 1000and the x-axis rotation angle θHSx for the lens 100, and a differencevalue erry between the y-axis rotation angle θYaw for the camera module1000 and the y-axis rotation angle θHSy for the lens 100. For example,the error value may be expressed as—(| errx+erry |).

Defocus_amount may be adjusted according to a voltage vz value. The AFcoil 260 may adjust the z-axis of the AF coil 260 according to the czvalue corresponding to the voltage vz. Accordingly, the z-axis of thelens 100 fixed to the lens holder 210 provided with the AF coil 260 maybe moved. Therefore, Defocus_amount can be adjusted.

The stabilization signal generator 1420 may further include a conversionunit Transpose configured to change the inferred stabilization data(+Xaxis, —Xaxis, +Yaxis, —Yaxis, +Zaxis, —Zaxis) into a voltage usablefor input of the VCM driver 130. The conversion unit Transpose convertsthe stabilization data into voltages (vx, vy, vz) using a conversionformula or a lookup table, and outputs the converted voltages (vx, vy,vz) as a control signal.

When the voltages (vx, vy, vz) are input, the VCM driver 130 may outputa control signal for compensating for vibration of the VCM actuator 200according to the stabilization data. These control signals may becurrents (cx, cy, cz) for controlling the VCM actuator 200.

When the current (cx, cy, cz) is input, the VCM actuator 200 operates tocompensate the position of the lens 100 according to the input current,thereby stabilizing the image and adjusting the focus.

The camera module 1000 according to various examples of the presentdisclosure may obtain tremor detection data (θYaw, θPitch, θHSx, θHSy)and temperature data Temp through a sensor, and defocus amount datadefocus amount from the lens focus data θHSz and the frequency componentof the image.

The camera module 1000 may output stabilization data using an ANN modeltrained to infer stabilization data for image stabilization and focusadjustment based on the obtained tremor detection data, temperaturedata, lens focus data, and defocus amount data.

Hereinafter, a method for image stabilization in the camera module willbe described with reference to FIG. 15 , which illustrates a method forimage stabilization in a camera module in an example of the presentdisclosure. Operations to be described later in the presented examplemay be performed by the processor 140 of the camera module 1000.

Referring to FIG. 15 , the processor 140 acquires image-related tremordetection data from two or more sensors (S1500). The two or more sensorsaccording to an example of the present disclosure may include two ormore of a gyro sensor, a Hall sensor, and a photo sensor. Here, thetremor detection data may include a signal detected by the x-axis andy-axis rotational movement of the gyro sensor and the x-axis and y-axisrotational movement of the Hall sensor.

The processor 140 outputs stabilization data using an ANN model trainedto output stabilization data for compensating for image shaking based onthe obtained tremor detection data (S1510).

Next, the processor 140 compensates for image shaking using the outputstabilization data (S1520).

FIG. 16 illustrates a neural processing unit according to the presentdisclosure.

The neural processing unit (NPU) shown in FIG. 16 is a processorspecialized to perform an operation for an artificial neural network.

An artificial neural network refers to a network of artificial neuronsthat multiplies and adds weights when multiple inputs or stimuli arereceived, and transforms and transmits the value added with anadditional deviation through an activation function. The trainedartificial neural network can be used to output inference results frominput data.

The NPU may be a semiconductor implemented as an electric/electroniccircuit. The electric/electronic circuit may include number ofelectronic devices (e.g., a transistor and a capacitor).

The NPU may include a processing element (PE) array 11000, an NPUinternal memory 12000, an NPU scheduler 13000, and an NPU interface14000. Each of the plurality of processing elements 11000, the NPUinternal memory 12000, the NPU scheduler 13000, and the NPU interface14000 may be a semiconductor circuit to which numerous transistors areconnected. Therefore, some transistors may be difficult to identify anddistinguish with the naked eye, and may be identified only by anoperation.

The NPU may be configured to infer a first model (e.g., Actor model).

For example, a specific circuit may operate as the plurality ofprocessing elements 11000, or may operate as the NPU scheduler 13000.The NPU scheduler 13000 may be configured to perform the function of thecontroller configured to control the artificial neural network inferenceoperation of the NPU.

The NPU may include a plurality of processing elements 11000, an NPUinternal memory 12000 configured to store an artificial neural networkmodel that can be inferred by the plurality of processing elements11000, and a NPU scheduler 13000 configured to control the plurality ofprocessing elements 11000 and the NPU internal memory 12000 based on thedata locality information or information about the structure of theartificial neural network model. Here, the artificial neural networkmodel may include information on data locality information or structureof the artificial neural network model. The artificial neural networkmodel may refer to an AI recognition model trained to perform a specificinference function.

The plurality of processing elements 11000 may perform an operation foran artificial neural network.

The NPU interface 14000 may communicate with various elements connectedto the NPU through a system bus, for example, a memory.

The NPU scheduler 13000 may be configured to control the operation ofthe plurality of processing elements 11000 for the inference operationof the neural processing unit and the sequence of the read operation andthe write operation of the NPU internal memory 12000.

The NPU scheduler 13000 may be configured to control the plurality ofprocessing elements 11000 and the NPU internal memory 12000 based on thedata locality information or information about the structure of theartificial neural network model.

The NPU scheduler 13000 may analyze the structure of the artificialneural network model to be operated in the plurality of processingelements 11000 or may receive the pre-analyzed information. For example,the data of the artificial neural network that can be included in anartificial neural network model may include at least a portion of nodedata (i.e., feature map) of each layer, arrangement data of layers,locality information or structure information, and weight data of eachconnection network (i.e., weight kernel) connecting nodes of each layer.The data of the artificial neural network may be stored in a memoryprovided inside the NPU scheduler 13000 or the NPU internal memory12000.

The NPU scheduler 13000 may schedule the operation sequence of theartificial neural network model to be performed by the NPU based on thedata locality information or the structure information of the artificialneural network model.

The NPU scheduler 13000 may acquire a memory address value in which thefeature map and weight data of the layer of the artificial neuralnetwork model are stored based on the data locality information or thestructure information of the artificial neural network model. Forexample, the NPU scheduler 13000 may obtain a memory address value inwhich the feature map and weight data of the layer of the artificialneural network model stored in the memory are stored. Therefore, the NPUscheduler 13000 may transmit the feature map and weight data of thelayer of the artificial neural network model to be driven from thememory and store it in the NPU internal memory 12000.

The feature map of each layer may have a corresponding memory addressvalue, respectively.

Each weight data may have a corresponding memory address value,respectively.

The NPU scheduler 13000 may schedule an operation sequence of theplurality of processing elements 11000 based on the data localityinformation or the information about the structure of the artificialneural network model, for example, the data locality information oflayout of layers of the artificial neural network model or theinformation about the structure of the artificial neural network model.

The NPU scheduler 13000 may schedule based on the data localityinformation or the information about the structure of the artificialneural network model so that the NPU scheduler may operate in adifferent way from a scheduling concept of a normal CPU. The schedulingof the normal CPU operates to provide the highest efficiency inconsideration of fairness, efficiency, stability, and reaction time.That is, the normal CPU schedules to perform the most processing duringthe same time in consideration of a priority and an operation time.

A conventional CPU uses an algorithm which schedules a task inconsideration of data such as a priority or an operation processing timeof each processing. In contrast, the NPU scheduler 13000 may determine aprocessing sequence based on the data locality information or theinformation about the structure of the artificial neural network model.

Moreover, the NPU scheduler 13000 may operate the NPU according to thedetermined processing sequence based on the data locality information orthe information about the structure of the artificial neural networkmodel and/or data locality information or information of a NPU. However,the present disclosure is not limited to the data locality informationor the information about the structure of the NPU.

NPU scheduler 13000 may be configured to store information about thedata locality information or structure of the artificial neural networkmodel.

That is, the NPU scheduler 13000 may determine the processing sequenceeven if only information on the data locality information or structureof the artificial neural network model is provided.

Furthermore, the NPU scheduler 13000 may determine the processingsequence of the NPU in consideration of the information on the datalocality information or structure of the artificial neural network modeland the data locality information or information on the structure of theNPU. In addition, it is also possible to optimize the processing of theNPU in the determined processing sequence.

The plurality of processing elements 11000 may refer to a configurationin which a plurality of processing elements PE1 to PE12 configured tocalculate the feature map and weight data of the artificial neuralnetwork are disposed. Each processing element may include a multiply andaccumulate (MAC) operator and/or an Arithmetic Logic Unit (ALU)operator. However, examples according to the present disclosure are notlimited thereto.

Each processing element may be configured to optionally further includean additional special function unit for processing the additionalspecial function.

For example, it is also possible for the processing element PE to bemodified and implemented to further include a batch-normalization unit,an activation function unit, an interpolation unit, and the like.

Although FIG. 16 illustrates a plurality of processing elements, it isalso possible to configure operators implemented as a plurality ofmultipliers and adder trees to be arranged in parallel by replacing theMAC in one processing element. In this case, the plurality of processingelements 11000 may be referred to as at least one processing elementincluding a plurality of operators.

The plurality of processing elements 11000 is configured to include aplurality of processing elements PE1 to PE12. The plurality ofprocessing elements PE1 to PE12 illustrated in FIG. 16 is merely anexample for convenience of description, and the number of the pluralityof processing elements PE1 to PE12 is not limited thereto. The size ornumber of the processing element array may be determined by the numberof the plurality of processing elements PE1 to PE12. The size of theprocessing element array may be implemented in the form of an N×Mmatrix. Here, N and M are integers greater than zero. The processingelement array may include N×M processing elements. That is, there may beat least one processing elements.

The size of the plurality of processing elements 11000 may be designedin consideration of the characteristics of the artificial neural networkmodel in which the NPU operates.

The plurality of processing elements 11000 may be configured to performfunctions such as addition, multiplication, and accumulation requiredfor an artificial neural network operation. In other words, theplurality of processing elements 11000 may be configured to perform amultiplication and accumulation (MAC) operation.

According to various examples of the present disclosure, the artificialneural network model may be trained based on the reinforcement learningtechnique.

According to various examples of the present disclosure, the tremordetection data may include a signal for detecting a change in theposition of the camera module and the lens.

According to various examples of the present disclosure, the artificialneural network model may be trained according to the tremor detectiondata such that an error value due to the image shaking approaches apredetermined value. Here, the error value may be based on a differencebetween the x-axis movement of the gyro sensor and the x-axis movementof the hall sensor and a difference between the y-axis movement of thegyro sensor and the y-axis movement of the hall sensor.

According to various examples of the present disclosure, the artificialneural network model may output a control signal for controlling amovement of a lens included in a camera module to compensate for theimage shaking by receiving the tremor detection data as an input.

According to various examples of the present disclosure, the artificialneural network model may output a control signal for controlling amovement of an image sensor included in a camera module to compensatefor the image shaking by receiving the tremor detection data as aninput.

According to various examples of the present disclosure, the trainedmodel may include a first model trained to infer the stabilization datain which the error value approaches the predetermined value based on thetremor detection data, and a second model trained to criticize theresult of the stabilization data.

According to various examples of the present disclosure, the artificialneural network model may simultaneously perform training for inferringstabilization data by inputting tremor detection data as an input andinferring the stabilization data.

According to various examples of the present disclosure, the artificialneural network model may include an input node to which tremor detectiondata is input, a hidden layer performing an AI operation (e.g.,convolutional operation), and an output node to output stabilizationdata.

According to various examples, the processor 140 may be configured tocollect the error value through the training and update the artificialneural network model using the collected error value.

According to various examples of the present disclosure, the cameramodule may include a temperature sensor for sensing a temperature andthe processor 140 may be configured to output the stabilization datausing the artificial neural network model based on the tremor detectiondata and a temperature data acquired through the temperature sensor.

According to various examples of the present disclosure, the tremordetection data may include a signal detected by an x-axis and y-axisrotational movement of a gyro sensor and an x-axis, y-axis and z-axisrotational movement of a hall sensor and the processor 140 may beconfigured to obtain a defocus amount data from a frequency component ofthe image, and output the stabilization data using the artificial neuralnetwork model based on the tremor detection data and the defocus amountdata.

According to various examples of the present disclosure, the artificialneural network model may use a modulation transfer function (MTF) dataof the image as a training data for training. The MTF data may be dataobtained by quantifying the amount of defocus.

The examples illustrated in the specification and the drawings aremerely provided to facilitate the description of the subject matter ofthe present disclosure and to provide specific examples to aid theunderstanding of the present disclosure and it is not intended to limitthe scope of the present disclosure. It is apparent to those of ordinaryskill in the art to which the present disclosure pertains in which othermodifications based on the technical spirit of the present disclosurecan be implemented in addition to the examples disclosed herein.

What is claimed is:
 1. A processor for stabilizing an image based onartificial intelligence (AI) comprising: at least one processing elementand at least one memory electrically coupled to the at least oneprocessing element, wherein the processor is configured to acquiretremor detection data, and output stabilization data for compensatingfor shaking of the image, wherein the tremor detection data is acquiredfrom two or more sensors, and wherein the stabilization data isoutputted using an artificial neural network (ANN) model trained tooutput the stabilization data based on the tremor detection data.
 2. Theprocessor of claim 1, wherein the ANN model is trained based onreinforcement learning.
 3. The processor of claim 1, wherein a virtualtremor detection data is provided for reinforcement learning for the ANNmodel.
 4. The processor of claim 2, wherein the virtual tremor detectiondata includes at least one of sitting pattern, walking pattern, runningpattern, a motion pattern of a ship, a motion pattern of a car and amotion pattern of a motorcycle.
 5. The processor of claim 1, wherein thetremor detection data includes a signal for detecting position change ofthe two or more sensors.
 6. The processor of claim 1, wherein the two ormore sensors include a gyro sensor and a Hall sensor.
 7. The processorof claim 1, wherein the tremor detection data includes a signal detectedby an x-axis and y-axis rotational movement of the two or more sensors.8. The processor of claim 1, wherein the processor further acquiresdefocus amount data of the image input to the ANN model.
 9. Theprocessor of claim 1, wherein the ANN model is a trained model so thatan error value due to the image shaking approaches a predetermined valuebased on training data for learning.
 10. The processor of claim 1,wherein the ANN model includes: an input node to which the tremordetection data is input; a hidden layer for performing an AI operationof the input node; and an output node to outputting the stabilizationdata.
 11. The processor of claim 1, wherein the processor is implementedas a system on chip (SoC) in which a central processing unit (CPU) and aneural processing unit (NPU) are integrated.
 12. The processor of claim1, wherein the processor is implemented as at least one of a centralprocessing unit (CPU), an application processor (AP), a micro processingunit (MPU), a micro controller unit (MCU), an image signal processor(ISP) or a neural processing unit (NPU).
 13. A device including: a lens;an image sensor coupled to the lens; and at least two or more sensorscoupled to the lens or the image sensor; and a processor configured toreceive tremor detection data, with respect to the lens or the imagesensor, from the at least two or more sensors to process a trainedartificial neural network (ANN) model so as to output stabilizationdata.
 14. The device of claim 13, further comprising: a VCM driverconfigured to receive the stabilization data from the processor andoutput a control signal; and a VCM actuator configured to compensate forposition of the lens or the image sensor based on the control signal.15. The device of claim 14, wherein the VCM driver is configured toconvert the stabilization data into the control signal using aconversion formula or a lookup table.
 16. The device of claim 14,wherein the VCM actuator is configured to control the position of thelens or the image sensor in x-axis and/or y-axis.
 17. A processorcomprising: a plurality of processing elements including a multiply andaccumulate operator and/or an arithmetic logic unit operator forperforming artificial intelligence operations, wherein the processor isconfigured to receive an image and tremor detection data, and generatestabilization data for compensating for shaking of the image, whereinthe stabilization data is inferred by a trained artificial neuralnetwork (ANN) model processed by the processor.
 18. The device of claim17, wherein the ANN model is trained by the tremor detection data orvirtual tremor detection data.
 19. The device of claim 17, wherein theprocessor is configured to further receive at least one of temperaturedata and defocus amount data so as to infer the stabilization data. 20.The device of claim 17, wherein the stabilization data is configured tobe input to a VCM driver to output a control signal to a VCM actuator;and wherein the control signal is configured to be input to the VCMactuator configured to compensate for shaking of the image.